Systems and methods for classifying activated T cells

ABSTRACT

Systems and methods for classifying and/or sorting T cells by activation state are disclosed. The system includes a cell classifying pathway, a single-cell autofluorescence image sensor, a processor, and a non-transitory computer-readable memory. The memory is accessible to the processor and has stored thereon a trained convolutional neural network and instructions. The instructions, when executed by the processor, cause the processor to: a) receive the autofluorescence intensity image; b) optionally pre-process the autofluorescence intensity image to produce an adjusted autofluorescence intensity image; c) input the autofluorescence intensity image or the adjusted autofluorescence intensity image into the trained convolutional neural network to produce an activation prediction for the T cell.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to, claims priority to, and incorporatedherein by reference for all purposes U.S. Provisional Patent ApplicationNo. 62/886,139, filed Aug. 13, 2019.

STATEMENT REGARDING FEDERALLY FUNDED RESEARCH

This invention was made with government support under CA205101 awardedby the National Institutes of Health. The government has certain rightsin the invention.

BACKGROUND

One new cancer treatment being studied is CAR T cell (Chimeric AntigenReceptor T cell) therapy. CAR T cell therapy uses a patient's own cellsand “re-engineers” them to fight cancer. It is a very complex treatment.Collecting and altering the cells is difficult, and CAR T cell therapyoften causes very severe side effects. At this time, it is only offeredat a few major cancer centers. To date, most of the patients treatedwith CAR T cells have been people with blood cancers.

The procedure starts with removing the patient's own T cells from theirblood and sending them to a lab where they are altered to produceproteins called chimeric antigen receptors (CARs) on the surface of thecells. These special receptors allow the T cells to help identify andattack cancer cells. The “super-charged” T cells are multiplied andgrown at the lab, then frozen and shipped back to the Hospital, wherethey re-inject these treated CAR T cells back into the patient's blood.

Current methods to determine T cell activation include flow cytometry,immunofluorescence imaging, and immunohistochemistry but these methodsrequire contrast agents and may require tissue or cell fixation. A needexists for systems and methods for classifying and/or sorting T cells byactivation state in a fashion that allows the sorted T cells to be usedin subsequent procedures, such as CAR T cell therapy.

SUMMARY

In an aspect, the present disclosure provides a T cell classifyingand/or sorting device. The device includes a cell analysis pathway, asingle-cell autofluorescence image sensor, a processor, and anon-transitory computer-readable medium. The cell analysis pathwayincludes an inlet, an observation zone, and outlet, and an optional cellsorter. The observation zone is coupled to the inlet downstream of theinlet and to the outlet upstream of the outlet. The observation zone isconfigured to present T cells for individual autofluorescenceinterrogation. The optional cell sorter has a sorter inlet and at leasttwo sorter outlets. The optional cell sorter is coupled to theobservation zone via the sorter inlet downstream of the observationzone. The optional cell sorter is configured to selectively direct acell from the sorter inlet to one of the at least two sorter outletsbased on a sorter signal. The single-cell autofluorescence image sensoris configured to acquire an autofluorescence intensity image of a T cellpositioned in the observation zone. The processor is in electroniccommunication with the single-cell autofluorescence image sensor. Thenon-transitory computer-readable medium is accessible to the processor.The non-transitory computer-readable medium has stored thereon a trainedconvolutional neural network and instruction. The instruction, whenexecuted by the processor, cause the processor to: a) receive theautofluorescence intensity image; b) optionally pre-process theautofluorescence intensity image to produce an adjusted autofluorescenceintensity image; c) input the autofluorescence intensity image or theadjusted autofluorescence intensity image into the trained convolutionalneural network to produce an activation prediction for the T cell.

In another aspect, the present disclosure provides a method ofcharacterizing T cell activation state. The method includes: a)optionally receiving a population of T cells having unknown activationstatus; b) acquiring an autofluorescence intensity image for a T cell ofthe population of T cells; c) optionally pre-processing theautofluorescence intensity image to provide an adjusted autofluorescenceintensity image; and d) identifying an activation status of the T cellbased on an activation prediction, wherein the activation prediction iscomputed using the autofluorescence intensity image or the adjustedautofluorescence intensity image as an input for a trained convolutionalneural network.

In a further aspect, the present disclosure provides a method of sortingand/or classifying T cells. The method includes: a) receiving apopulation of T cells having unknown activation status; b) acquiring anautofluorescence intensity image for each T cell of the population of Tcells, thereby resulting in a set of autofluorescence intensity images;c) optionally pre-processing the autofluorescence intensity images ofthe set of autofluorescence intensity images to provide a set ofadjusted autofluorescence intensity images; and either: d1) physicallyisolating a first portion of the population of T cells from a secondportion of the population of T cells based on an activation prediction,wherein each T cell of the population of T cells is placed into thefirst portion when the activation prediction exceeds a predeterminedthreshold and into the second portion when the activation prediction isless than or equal to the predetermined threshold; or d2) generating areport including the activation prediction, the report optionallyidentifying a proportion of the population of T cells having theactivation prediction that exceeds the predetermined threshold. Theactivation prediction is computed using the autofluorescence intensityimage from the set of autofluorescence intensity images or the adjustedautofluorescence intensity image from the set of adjustedautofluorescence intensity images corresponding to a given T cell as aninput for a trained convolutional neural network.

In yet another aspect, the present disclosure provides a method ofadministering activated T cells to a subject in need thereof. The methodincludes: a) the method of sorting and/or classifying T cells asdescribed herein; and b) introducing the first portion (or any portionincluding a sufficient amount of activated T cells) to the subject.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a flowchart illustrating a method, in accordance with anaspect of the present disclosure.

FIG. 2 is a flowchart illustrating a method, in accordance with anaspect of the present disclosure.

FIG. 3 is a block diagram of a device, in accordance with an aspect ofthe present disclosure.

FIG. 4 is one exemplary T cell image data processing workflow, asdescribed in the present disclosure and Example 1.

FIG. 5 is a model summary per donor for three different evaluationmetrics, as described in Example 1. The line graphs display theclassifiers' performance across donors. The bar graphs display thenumber of activated and quiescent images for each donor, which affectsthe classifiers' performance.

FIG. 6 shows receiver operating characteristic (ROC) curves for eachtype of classifier and donor, as described in Example 1. The gray barsto the right display the number of activated and quiescent images foreach donor.

FIG. 7 shows precision recall (PR) curves for each type of classifierand donor, as described in Example 1. The gray bars to the right displaythe number of activated and quiescent images for each donor.

FIG. 8 is a performance comparison of fine-tuning a different number oflayers and the pre-trained CNN off-the-shelf model, as described inExample 1.

FIG. 9 includes 2D representations of T cell features extracted from thepre-trained CNN with fine-tuning, as described in Example 1. Dimensionsare reduced from 2048 using UMAP. Data points with thick outlinesindicate incorrect cell activity state predictions.

FIG. 10 shows saliency maps of randomly-selected cell images from donor1 (scale bar: 10 μm), as described in Example 1. The backpropagation andguided backpropagation rows show two different techniques for generatingsaliency maps from the same T cell images in the first row.

FIG. 11 shows a 5×4 nested cross-validation scheme, as discussed inExample 1. For each test donor (blue), we used an inner cross-validationloop to optimize the hyper-parameters. We trained a model for eachhyper-parameter combination using the training donors' augmented images(yellow) and selected the hyper-parameters that performed best on thevalidation donor's images (green). The validation donor is sometimesreferred to as a tuning donor in cross-validation. Then, we trained afinal model for each test donor using the selected hyper-parameters.

FIG. 12 is a visual representation of fine-tuning of the Inception v3CNN to predict T cell activity, as described in Example 1. The genericimage examples are adapted from ImageNet.

DETAILED DESCRIPTION

Before the present invention is described in further detail, it is to beunderstood that the invention is not limited to the particularembodiments described. It is also understood that the terminology usedherein is for the purpose of describing particular embodiments only, andis not intended to be limiting. The scope of the present invention willbe limited only by the claims. As used herein, the singular forms “a”,“an”, and “the” include plural embodiments unless the context clearlydictates otherwise.

Specific structures, devices and methods relating to modifyingbiological molecules are disclosed. It should be apparent to thoseskilled in the art that many additional modifications beside thosealready described are possible without departing from the inventiveconcepts. In interpreting this disclosure, all terms should beinterpreted in the broadest possible manner consistent with the context.Variations of the term “comprising” should be interpreted as referringto elements, components, or steps in a non-exclusive manner, so thereferenced elements, components, or steps may be combined with otherelements, components, or steps that are not expressly referenced.Embodiments referenced as “comprising” certain elements are alsocontemplated as “consisting essentially of” and “consisting of” thoseelements. When two or more ranges for a particular value are recited,this disclosure contemplates all combinations of the upper and lowerbounds of those ranges that are not explicitly recited. For example,recitation of a value of between 1 and 10 or between 2 and 9 alsocontemplates a value of between 1 and 9 or between 2 and 10.

As used herein, the terms “activated” and “activation” refer to T cellsthat are CD3+, CD4+, and/or CD8+.

As used herein the term “convolutional neural network” refers to a typeof deep neural network typically consisting of a series of convolutionallayers that classifies images. Convolutional neural networks operate onspatially-local regions of input images to recognize patterns in thoseregions. Convolutional neural networks can include fully-connectedlayers and other types of layers in addition to the convolutionallayers.

As used herein, the term “FAD” refers to flavin adenine dinucleotide.

As used herein, the term “memory” includes a non-volatile medium, e.g.,a magnetic media or hard disk, optical storage, or flash memory; avolatile medium, such as system memory, e.g., random access memory (RAM)such as DRAM, SRAM, EDO RAM, RAMBUS RAM, DR DRAM, etc.; or aninstallation medium, such as software media, e.g., a CD-ROM, or floppydisks, on which programs may be stored and/or data communications may bebuffered. The term “memory” may also include other types of memory orcombinations thereof.

As used herein, the term “NAD(P)H” refers to reduced nicotinamideadenine dinucleotide and/or reduced nicotinamide dinucleotide phosphate.

As used herein, the term “processor” may include one or more processorsand memories and/or one or more programmable hardware elements. As usedherein, the term “processor” is intended to include any of types ofprocessors, CPUs, GPUs, microcontrollers, digital signal processors, orother devices capable of executing software instructions.

As used herein, the term “training” refers to a process that provideslabeled data to a classification algorithm to learn to map input data toa category.

As used herein, the term “pre-training” refers to training a classifieron an initial dataset that is a different and typically larger datasetthan the target dataset. As used herein, pre-training initializes aclassifier so that it can by trained faster or more accurately on thetarget dataset.

As used herein, the term “fine-tuning” refers to use of the pre-trainedmodel weights to initialize parameters of a new model and then train anew model on the target dataset.

The various aspects may be described herein in terms of variousfunctional components and processing steps. It should be appreciatedthat such components and steps may be realized by any number of hardwarecomponents configured to perform the specified functions.

Methods

This disclosure provides a variety of methods. It should be appreciatedthat various methods are suitable for use with other methods. Similarly,it should be appreciated that various methods are suitable for use withthe systems described elsewhere herein. When a feature of the presentdisclosure is described with respect to a given method, that feature isalso expressly contemplated as being useful for the other methods andsystems described herein, unless the context clearly dictates otherwise.

Referring to FIG. 1, the present disclosure provides a method 100 ofsorting T cells. At process block 102, the method 100 includes receivinga population of T cells having unknown activation status. The populationof T cells can itself be contained within a broader population of cellsthat includes some cells that are not T cells, such as red blood cellsand the like. At process block 104, the method 100 includes acquiring anautofluorescence intensity image for each T cell of the population of Tcells, thereby resulting in a set of autofluorescence intensity images.At optional process block 105, the method 100 optionally includespre-processing the autofluorescence intensity images of the set ofautofluorescence intensity images to provide a set of adjustedautofluorescence intensity images. At optional process block 106, themethod 100 optionally includes physically isolating a first portion ofthe population of T cells from a second portion of the population of Tcells based on an activation prediction. At optional process block 108,the method 100 optionally includes generating a report including anactivation prediction.

The autofluorescence intensity image acquired at process block 104 canbe acquired in a variety of ways, as would be understood by one havingordinary skill in the spectroscopic arts with knowledge of thisdisclosure and their own knowledge from the field. The specific way inwhich the autofluorescence intensity image is acquired is not intendedto be limiting to the scope of the present invention, so long as theautofluorescence intensity images necessary for the methods describedherein can be suitably measured, estimated, or determined in anyfashion. One example of a suitable autofluorescence intensity imageacquisition is described below in the examples section.

The optional pre-processing of process block 105 can include variousimage processing steps for providing better consistency of images forintroduction to the convolutional neural network. The pre-processing caninclude cropping the autofluorescence intensity image, padding theautofluorescence intensity image (i.e., adding black pixels to a side ofthe image to artificially increase the image's size), rotating theautofluorescence intensity image, reflecting the autofluorescenceintensity image (i.e., taking a mirror image of the original about agiven axis), or a combination thereof.

The optional physically isolating of process block 106 is in response toan activation prediction determined from the acquired autofluorescenceintensity image. If the activation prediction exceeds a predeterminedthreshold for a given T cell, then that T cell is placed into the firstportion. If the activation prediction is less than or equal to thepredetermined threshold for the given T cell, then that T cell is placedinto the second portion. The result of this physical isolation is thatthe first portion of the population of T cells is significantly enrichedin activated T cells, whereas the second portion of the population of Tcells is significantly depleted of activated T cells.

In some cases, the physically isolating of process block 106 can includeisolating cells into three, four, five, six, or more portions. In thesecases, the different portions will be separated by a number ofpredetermined thresholds that is one less than the number of portions(i.e., three portions=two predetermined thresholds). The portion whoseactivation prediction exceeds all of the predetermined thresholds (i.e.,exceeds the highest threshold) contains the greatest concentration ofactivated T cells. The portion whose activation prediction fails toexceed any of the predetermined thresholds (i.e., fails to exceed thelowest threshold) contains the lowest concentration of activated Tcells. Using multiple predetermined thresholds can afford thepreparation of portions of the population of T cells that have extremelyhigh or extremely low concentrations of activated T cells.

The optional generating a report of process block 108 can include anyform of report generation known to be useful in the medical arts,including but not limited to, generating a digital report, a displayshowing results, printing a physical hard copy of a report The reportoptionally identifies a proportion of the population of T cells havingthe activation prediction that exceeds the predetermined threshold.

The activation status is computed using a convolutional neural network.The convolutional neural network can be pre-trained with images that arenot fluorescence intensity images of cells, and then it can befine-tuned with images that are fluorescence intensity images of T cellswith a known activation state.

The pre-training can include training with standard images of objectsthat are visible to the human eye (i.e., a neural network pre-trained toidentify dogs as dogs, cats as cats, humans as humans, etc.). Variouscommercially-available neural networks come pre-trained in this fashion.For example, the Inception v3 convolutional neural network withpre-trained ImageNet weights discussed below utilizes this type ofpre-training.

The training of the convolutional neural network involves inputting anumber of autofluorescence intensity images for T cells with a knownactivation state. In some cases, the only input for the training of theconvolutional neural network is the series of autofluorescence intensityimages for T cells with a known activation state. After this traininghas occurred, the result is a trained convolutional neural network,which is ready to receive autofluorescence intensity images of T cellsthat have an unknown activation state and to make an activationprediction. The convolutional neural network can be trained with atleast 100 images, at least 200 images, at least 500 images, at least1000 images, at least 2500 images, at least 5000 images, or more imagesof T cells having known activation states. The trained convolutionalneural network can include at least 5 layers, at least 6 layers, atleast 7 layers, at least 8 layers, at least 9 layers, or at least 10layers. The trained convolutional neural network can include at most 100layers, at most 50 layers, at most 20 layers, at most 17 layers, at most15 layers, at most 14 layers, at most 13 layers, or at most 12 layers.The convolutional neural network can include segmenting of theautofluorescence intensity images.

In some cases, the trained convolutional neural network can beinstrument-specific. It should be appreciated that thisinstrument-specificity can encompass specificity for a given specificinstrument or specificity for a given model of a specific instrument.

In some cases, the trained convolutional neural network can bepatient-specific. In these cases, the convolutional neural network is atleast partially trained with T cells having a known activation statethat come from similar patients. Patient similarity can be assessedbased on patient age, sex, disease subtype, or other characteristics.

In some cases, the pre-training and the training can both be donewithout utilizing a cell size attribute (e.g., diameter), cellmorphology, or either as inputs.

The method 100 can sort CD4+, CD3+, and/or CD8+ T cells based onactivation status.

The method 100 can provide surprising accuracy of classifying T cells asactivated. The accuracy can be at least 85%, at least 87.5%, at least90%, at least 92.5%, at least 95%, at least 96%, at least 97%, or atleast 98%. One non-limiting example of measuring the accuracy includesexecuting the method 100 on a given cell with unknown activation stateand then using one of the traditional methods for determining activationstate (which will typically be a destructive method) for a number ofcells that is statistically significant.

The method 100 can be performed without the use of a fluorescent labelfor binding the T cell. The method 100 can be performed withoutimmobilizing the T cell.

In some cases, the method 100 can include a step for identifying outlierimages. In particular, if the image contains no cells at all or containscells that are not T cells (e.g., a red blood cell), then those imagescan be discarded and any cells corresponding to those images can also bediscarded.

Referring to FIG. 2, the present disclosure provides a method 200 ofadministering activated T cells to a subject in need thereof. At processblock 202, the method 200 includes the method 100 described above, whichresults in a first portion of the population of T cells enriched foractivation or a population of T cells for which a report has beengenerated regarding activation status. At optional process block 204,the method 200 optionally includes modifying the first portion of thepopulation of T cells (in the case where sorting did occur) or thepopulation of T cells (in the case where the sorting did not occur). Atprocess block 206, the method 200 includes administering the firstportion of the population of T cells or the population of T cells to thesubject.

The T cells can be harvested from the subject to which they areadministered prior to sorting. The sorted T cells or the population of Tcells can be either directly introduced to the subject or can undergoadditional processing prior to introduction to the subject. In one case,the sorted T cells or the population of T cells can be modified tocontain chimeric antigen receptors (CARs).

In some cases, the method of administering activated T cells can includeadministering an unsorted population of T cells for which the proportionof activated T cells is known to be above a given threshold (i.e., ifgreater than a given percentage of a population of T cells has anactivation prediction that exceeds the predetermined threshold, then theentire population can be administered to the subject).

The methods described herein provided surprising results to theinventors in at least three ways. First, it was not clear at the outsetwhether the methods would be effective at distinguishing activatedversus quiescent T cells, because it was not clear that autofluorescenceintensity images could classify accurately. The efficacy itself wassurprising, and the quality of the classification achieved by themethods was even more surprising. Second, the inventors expected thattraining models using features quantified from the images, such as cellsize and/or cell morphologies, in the input to classification algorithmswould improve classification, because these features are related toactivation and common features for cell state classification in themicroscopy domain. It was surprisingly discovered that using thisinformation leads to worse classification than using autofluorescenceintensity images. Third, the degree of classification accuracy achievedwas surprising. The classification accuracy of upward of 85-95% orbetter that is achieved using a convolutional neural network trainedonly with autofluorescence images (this does not exclude pre-trainingusing other images) and with autofluorescence images as the lone inputis surprising.

Systems

This disclosure also provides systems. The systems can be suitable foruse with the methods described herein. When a feature of the presentdisclosure is described with respect to a given system, that feature isalso expressly contemplated as being combinable with the other systemsand methods described herein, unless the context clearly dictatesotherwise.

Referring to FIG. 3, the present disclosure provides a T cell sortingdevice 300. The device 300 includes a cell analysis pathway 302. Thecell analysis pathway 302 includes an inlet 304, an observation zone306, and an outlet 305. The device 300 optionally includes a cell sorter308. The observation zone 306 is coupled to the inlet 304 downstream ofthe inlet 304 and is coupled to the outlet 305 upstream of the outlet305. The device 300 also includes a single-cell autofluorescence imagesensor 310. The device 300 includes a processor 312 and a non-transitorycomputer-readable medium 314, such as a memory.

The inlet 304 can be any nanofluidic, microfluidic, or other cellsorting inlet. A person having ordinary skill in the art of fluidics hasknowledge of suitable inlets 304 and the present disclosure is notintended to be bound by one specific implementation of an inlet 304.

The outlet 305 can be nanofluidic, microfluidic, or other cell sortingoutlet. A person having ordinary skill in the art of fluidics hasknowledge of suitable outlets 305 and the present disclosure is notintended to be bound by one specific implementation of an outlet 305.

The observation zone 306 is configured to present T cells for individualautofluorescence interrogation. A person having ordinary skill in theart has knowledge of suitable observation zones 306 and the presentdisclosure is not intended to be bound by one specific implementation ofan observation zone 306.

The optional cell sorter 308 has a sorter inlet 316 and at least twosorter outlets 318. The cell sorter is coupled to the observation zone306 via the sorter inlet 316 downstream of the observation zone 306. Thecell sorter 308 is configured to selectively direct a cell from thesorter inlet 316 to one of the at least two sorter outlets 318 based ona sorter signal.

The inlet 304, observation zone 306, outlet 305, and optional cellsorter 308 can be components known to those having ordinary skill in theart to be useful in flow sorters, including commercial flow sorters. Thecell analysis pathway can further optionally include a flow regulator,as would be understood by those having ordinary skill in the art. Theflow regulator can be configured to provide flow of cells through theobservation zone at a rate that allows the single-cell autofluorescenceimage sensor 310 to acquire the autofluorescence intensity image. Auseful review of the sorts of fluidics that can be used in combinationwith the present disclosure is Shields et al., “Microfluidic cellsorting: a review of the advances in the separation of cells fromdebulking to rare cell isolation,” Lab Chip, 2015 Mar. 7; 15(5):1230-49, which is incorporated herein by reference in its entirety.

The single-cell autofluorescence image sensor 310 can be any detectorsuitable for acquiring single-cell autofluorescence images as understoodby those having ordinary skill in the optical arts. It should beappreciated that these images need not be acquired by acquiring allpixels simultaneously, so autofluorescence images acquired by point-and/or line-scanning methods are expressly contemplated. Examples ofsuitable single-cell autofluorescence image sensors 310 include, but arenot limited to, a camera, a photodiode array, a streak camera, a chargecapture device array, a photodiode, an avalanche photodiode, aphotomultiplier tube, combinations thereof, and the like.

The single-cell autofluorescence image sensor 310 can be directly (i.e.,the processor 312 communicates directly with the detector 310 andreceives the signals) or indirectly (i.e., the processor 312communicates with a sub-controller that is specific to the sensor 310and the signals from the sensor 310 can be modified or unmodified beforesending to the processor 312) controlled by the processor 312.Fluorescence intensity images can be acquired by known imaging methods.The device 300 can include various optical filters tuned to isolateautofluorescence signals of interest. The optical filters can be tunedto the autofluorescence wavelengths of NAD(P)H and/or FAD.

The device 300 can optionally include a light source 320 for opticallyexciting the cells to initiate autofluorescence. Suitable light sources320 include, but are not limited to, lasers, LEDs, lamps, filteredlight, fiber lasers, and the like. The light source 320 can becontinuous wave. The light source 320 can be pulsed, which includessources that are naturally pulsed and continuous sources that arechopped or otherwise optically modulated with an external component. Thelight source 320 can provide pulses of light having a full-width at halfmaximum (FWHM) pulse width of between 1 fs and 10 ns. In some cases, theFWHM pulse width is between 30 fs and 1 ns. The light source 320 canemit wavelengths that are tuned to the absorption of NAD(P)H and/or FAD.

The single-cell autofluorescence image sensor 310 can be configured toacquire the autofluorescence dataset at a repetition rate of between 1kHz and 20 GHz. In some cases, the repetition rate can be between 1 MHzand 1 GHz. In other cases, the repetition rate can be between 20 MHz and100 MHz. The light source 320 can be configured to operate at theserepetition rates.

The device 300 can optionally include a cell size measurement tool 322.The cell size measurement tool 322 can be any device capable ofmeasuring the size of cells, including but not limited to, an opticalmicroscope. In some cases, the single-cell autofluorescence image sensor310 and the cell size measurement tool 322 can be integrated into asingle optical subsystem. While some aspects of the methods describedherein can operate by not utilizing the cell size as an input to theconvolutional neural network, it may be useful to measure the cell sizefor other purposes.

The processor 312 is in electronic communication with the detector 310and the cell sorter 308. The processor 312 is also in electroniccommunication with, when present, the optional light source 320 and theoptional cell size measurement tool 322.

The non-transitory computer-readable medium 314 has stored thereoninstructions that, when executed by the processor, cause the processorto execute at least a portion of the methods described herein. Thetrained convolutional neural network can be stored in the non-transitorycomputer-readable medium 314. The non-transitory computer-readablemedium 314 can be local to the device 300 or can be remote from thedevice, so long as it is accessible by the processor 312.

The device 300 can be substantially free of fluorescent labels (i.e.,the cell analysis pathway 302 does not include a region for mixing thecell(s) with a fluorescent label). The device 300 can be substantiallyfree of immobilizing agents for binding and immobilizing T cells.

Example 1

Cell Preparation and Imaging

This study was approved by the Institutional Review Board of theUniversity of Wisconsin-Madison (#2018-0103). Informed consent wasobtained from all donors. The NAD(P)H intensity images in this studywere created from a subset of the NAD(P)H fluorescence lifetime imagesacquired in Walsh, A. et al. Label-free Method for Classification of Tcell Activation. bioRxiv (2019); DOI 10.1101/536813; (“Walsh et al.”),which is incorporated herein in its entirety by reference. CD3 and CD8 Tcells were isolated using negative selection methods (RosetteSep,StemCell Technologies) from the peripheral blood of 6 healthy donors (3male, 3 female, mean age=26). The T cells were divided into quiescentand activated groups, where the activated group was stimulated with atetrameric antibody against CD2, CD3, and CD28 (StemCell Technologies).T cell populations were cultured for 48 hours at 37 C, 5% CO2, and 99%humidity.

NAD(P)H intensity images were created by integrating the photon countsof fluorescence lifetime decays at each pixel within the fluorescencelifetime images acquired, as described by Walsh et al. Briefly, imageswere acquired using an Ultima (Bruker Fluorescence Microscopy)two-photon microscope coupled to an inverted microscope body (TiE,Nikon) with an Insight DS+ (Spectra Physics) as the excitation source. A100× objective (Nikon Plan Apo Lambda, NA 1.45), lending an approximatefield of view of 110 μm, was used in all experiments with the lasertuned to 750 nm for NAD(P)H two-photon excitation and a 440/80 nmbandpass emission filter in front of a GaAsP photomultiplier tube (PMT;H7422, Hamamatsu). Images were acquired for 60 seconds with a laserpower at the sample of 3.0-3.2 mW and a pixel dwell time of 4.6 μs.Grayscale microscopy images were labeled with a deidentified donor IDand T cell activity state according to the culture conditions: quiescentfor T cells not exposed to the activating antibodies or activated for Tcells exposed to the activating antibodies.

Image Processing

We segmented cell images using CellProfiler, which is described inCarpenter, A. E. et al. CellProfiler: image analysis software foridentifying and quantifying cell phenotypes. Genome Biol. 7, R100(2006); DOI 10.1186/gb-2006-7-10-r100; which is incorporated herein inits entirety by reference. Each cell was cropped according to thebounding box of its segmented mask. Cell short NAD(P)H lifetime was usedto filter out other visually indistinguishable cells (e.g., red bloodcells) by removing cells with a mean fluorescence lifetime less than 200ps. To remove very dim images and images containing no cells, we furtherfiltered the segmented images by thresholding the combination of imageentropy and total intensity. The segmented images were removed from thedataset if their entropy was less than 4 or if their entropy was lessthan 4.7 and their intensity was less than 3500. The threshold valueswere chosen based on the distribution of entropy and intensity with aGaussian approximation. This filter was conservative. We manuallyinspected the removed images to ensure none of them contained T cells.

Because the classifiers that used image pixels as input required uniformsize and some required square images, we padded all activated andquiescent cell images with black borders. The padding size of 82×82 waschosen based on the largest image in the dataset after removingextremely large outliers. Also, we augmented the dataset by rotatingeach original image by 90; 180; and 270 degrees and flipping ithorizontally and vertically. We implemented this image processingpipeline using the Python package OpenCV. Bradski, G. The OpenCVLibrary. Dr. Dobb's J. Softw. Tools (2000), which is incorporated hereinin its entirety by reference.

Nested Cross-Validation

We trained and evaluated eight classifiers of increasing complexity(Table 1). We used the same leave-one-donor-out test principle tomeasure the performance of all models. For example, when using donor 1as the test donor, the frequency classifier counts the positiveproportion among all images in the augmented dataset from donor 2, 3, 5,and 6. Then, it uses this frequency to predict the activity for allunaugmented images from donor 1. By testing in this way, theclassification result tells us how well each model performs on imagesfrom new donors. Donor 4 was not included in this cross-validationbecause we randomly selected it as a complete hold-out donor. All imagesfrom donor 4 were only used after hyper-parameter tuning and modelselection as a final independent test to assess the generalizability ofour pipeline to a new donor.

TABLE 1 Model Description Frequency Classifier Predict class probabilityusing the class fre- quencies in the training set. Logistic RegressionRegularized logistic regression model fitted with with Pixel Intensitythe image pixel intensity matrix (82 × 82). Regularization power λ of l₁penalty is tuned. Logistic Regression Regularized logistic regressionmodel fitted with with Total Intensity two numerical values: image totalintensity and and Size cell mask size. Regularization power λ of l₁penalty is tuned. Logistic Regression Regularized logistic regressionmodel fitted with with CellProfiler 123 features extracted fromCellProfiler related to Features intensity, texture, and area.Regularization power λ of l₁ penalty is tuned. One-layer Fully Fullyconnected one-hidden-layer neural network Connected Neural with pixelintensity as input. Number of neurons, Network learning rate, and batchsize are tuned. LeNet CNN CNN with the LeNet architecture with pixelintensity as input. Learning rate and batch size are tuned. Pre-trainedCNN Freeze layers of a pre-trained Inception v3 CNN. Off-the-shelf ModelTrain a final added layer from scratch with extract- ed off-the-shelffeatures. Learning rate and batch size are tuned. Pre-trained CNNFine-tune the last n layers of a pre-trained with Fine-tuning Inceptionv3 CNN. The layer number n, learning rate, and batch size are tuned.

Following the leave-one-donor-out test principle, we wanted theselection of the optimal hyper-parameters to be generalizable to newdonors as well. Therefore, we applied a nested cross-validation scheme(FIG. 11). For each test donor, within the inner loop we performed4-fold cross-validation to measure the average performance of eachhyper-parameter combination (grid search). Each fold in the inner loopcross-validation corresponds to one donor's augmented images. The outercross-validation loop used the selected hyper-parameters from the innerloop cross-validation to train a new model with the four other donors'augmented images. We evaluated the trained model on the outer loop testdonor. For models requiring early stopping, we constructed an earlystopping set by randomly sampling one-fourth of the unaugmented imagesfrom the training set and removing their augmented copies. Then,training continued as long as the performance on images in the earlystopping set improved. Similarly, we did not include augmented images inthe validation set or the test set.

No single evaluation metric can capture all the strengths and weaknessesof a classifier, especially because our dataset was class imbalanced andnot skewed in the same way for all donors. Therefore, we consideredmultiple evaluation metrics in the outer loop. Accuracy measures thepercentage of correct predictions. It is easy to interpret, but it doesnot necessarily characterize a useful classifier. For example, whenpositive samples are rare, a trivial classifier that predicts allsamples as negative yields high accuracy. Precision and recall(sensitivity), on the other hand, consider the costs of false positiveand false negative predictions, respectively. Graphical metrics like thereceiver operator characteristic (ROC) curve and precision recall (PR)curve avoid setting a specific classification threshold. We used areaunder the curve (AUC) to summarize ROC curves and average precision forthe PR curves. The ROC curve is independent of the class distribution,while the PR curve is useful when the classes are imbalanced. See,Lever, J., Krzywinski, M. & Altman, N. Points of Significance:Classification evaluation. Nat. Methods 13, 603-604 (2016); DOI10.1038/nmeth.3945; which is incorporated herein in its entirety byreference. For this reason, we used mean average precision of the innerloop 4-fold cross-validation to select optimal hyper-parameters.

During the nested cross-validation, we trained the LeNet CNN andpre-trained CNN with fine-tuning using GPUs. These jobs ran on GTX 1080,GTX 1080 Ti, K40, K80, P100, or RTX 2080 Ti GPUs. All other models weretrained using CPUs.

Linear Classifiers

We used a trivial frequency classifier as a baseline model. This modelcomputes the positive sample percentage in the training set. Then, ituses this frequency as a positive class prediction score (between 0and 1) for all samples in the test set.

Logistic regression with Lasso regularization is a standard andinterpretable statistical model used to classify microscopy images. See,Pavillon, N., Hobro, A. J., Akira, S. & Smith, N. I. Noninvasivedetection of macrophage activation with single-cell resolution throughmachine learning. Proc. Natl. Acad. Sci. 115, E2676-E2685 (2018); DOI10.1073/pnas.1711872115, which is incorporated herein in its entirety byreference. The Lasso approach reduces the number of effective parametersby shrinking the parameters of less predictive features to zero. Thesefeatures are ignored when making a new prediction. We fitted and testedthree Lasso logistic regression models with different types of featuresusing the Python package scikit-learn. See, Pedregosa, F. et al.Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12,2825-2830 (2011), which is incorporated herein in its entirety byreference. An image intensity matrix with dimension 82×82 and valuesfrom 0 to 255, reshaped into a vector with length 6,724, was used to fitthe first model. The second model was trained with two scalar features,cell size and image total intensity, where cell size was computed usingthe pixel count in the cell mask generated by CellProfiler. The lastmodel used 123 features relating to cell intensity, texture, and area,which were extracted from cell images using a CellProfiler pipeline withmodules MeaureObjectSizeShape, MeasureObjectIntensity, and MeasureTexture. The Lasso regularization parameter λ was tuned for all threeclassifiers with nested cross-validation (Table 2). We also appliedinverse class frequencies in the training data as class weights toadjust the imbalanced dataset.

TABLE 2 Hyper- Model parameter Candidate Logistic Regres- λ 0.001, 0.01,0.1, 1, 10, sion Models 100, 1000, 10000 One-layer Fully Learning Rate0.1, 0.01, 0.001, 0.0001, 0.00001 Connected Neural Batch Size 8, 16, 32,64 Network Neuron Number 16, 64, 128, 512, 1024 LeNet Cnn Learning Rate0.1, 0.01, 0.001, 0.0001, 0.00001 Batch Size 8, 16, 32, 64 Pre-trainedCNN Learning Rate 0.01, 0.001, 0.0001, 0.00001 Off-the-shelf Batch Size8, 16, 32, 64 Model Pre-trained CNN Learning Rate 0.01, 0.001, 0.0001,0.00001 with Fine-tuning Batch Size 8, 16, 32, 64 n 1, 2, 3, 4, 5, 6, 7,8, 9, 10, 11

Simple Neural Network Classifiers

We developed a simple fully-connected neural network with one hiddenlayer using the Python package Keras with the TensorFlow backend. See,Chollet, F. & others. Keras (2015); and Martin Abadi et al. TensorFlow:Large-Scale Machine Learning on Heterogeneous Systems (2015), both ofwhich are incorporated herein in their entirety by reference. The inputlayer uses the image pixel matrix with dimension 82×82. Networkhyper-parameters—hidden neuron numbers, learning rate, and batchsize—were tuned using nested cross-validation (Table 2). Thecross-entropy loss function was weighted according to the classdistribution in the training set.

Also, we trained a CNN with the LeNet architecture (see, Lecun, Y.,Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied todocument recognition. Proc. IEEE 86, 2278-2324 (1998); DOI10.1109/5.726791, which is incorporated herein in its entirety byreference) with randomly initialized weights (no pre-training). TheLeNet architecture has two convolutional layers and two pooling layers.We used the default number of neurons specified in the original paper ineach layer. The input layer was modified to support 82×82 one-channelimages, so we could train this network with image pixel intensities.Similar to the fully-connected neural network, we used nestedcross-validation to tune the learning rate and batch size (Table 2) andapplied class weighting. We used early stopping with a patience of 10for both models, which means we stopped training if the loss functionfailed to improve on the early stopping set in 10 consecutive epochs.

Pre-Trained CNN Classifiers

We developed a transfer learning classifier that uses the Inception v3CNN with pre-trained ImageNet weights. See, Szegedy, C., Vanhoucke, V.,Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the Inception Architecturefor Computer Vision; arXiv:1512.00567 [cs] (2015); and Deng, J. et al.ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 (2009),both of which are incorporated herein in their entirety by reference.Instead of training the whole network end-to-end from scratch, we tookadvantage of the pre-trained weights by extracting and modelingoff-the-shelf features or fine-tuning the last n Inception modules,where n was treated as a hyper-parameter (FIG. 12). Inception modulesare mini-networks that constitute the overall Inception v3 architecture.The first approach is a popular practice for transfer learning withInception v3. We freeze the weights of all layers before the outputlayer and use them to extract generic image characteristics. Then, wetrain a light-weight classifier from scratch, specifically a neuralnetwork with an average pooling layer and a fully connected hidden layerwith 1024 neurons, using these off-the-shelf features. We refer to thismodel as the pre-trained CNN off-the-shelf model. An alternative is tofix some earlier layers and fine-tune the higher-level n layers byinitializing them with pre-trained weights and continuing training on anew dataset. For this model, we modified the output layer to supportbinary classification, and we did not add new layers. In addition, weused the nested cross-validation scheme to optimize n along with thelearning rate and batch size (Table 2), creating the pre-trained CNNwith fine-tuning.

To implement these two pre-trained CNN models, we resized the paddedcell images with bilinear interpolation to fit the input layer dimension(299×299×3) and generated three-channel images by merging three copiesof the same grayscale image. For the pre-trained CNN with fine-tuning,we first used the resized cell images to generate intermediate features(“bottlenecks”). Then, we used these features to fine-tune asub-network. This approach significantly shortened the training time.Finally, we used class weighting and early stopping with a patience of10 for both models. We implemented these two models using Keras with theTensorFlow backend.

Pre-Trained CNN Interpretation

We implemented multiple approaches for interpreting the pre-trainedCNNs. Computing classification confidence on misclassified images canhelp us understand why classifiers make certain errors. The Softmaxscore is sometimes used as a confidence prediction. Softmax is afunction that maps the output real-valued number (Logit) from a neuralnetwork into a score between 0 and 1, which is then used to make aclassification as a class probability. However, using the Softmax scorefrom a neural network as a confidence calibration does not match thereal accuracy. Therefore, we used temperature scaling to bettercalibrate the predictions. See, Guo, C., Pleiss, G., Sun, Y. &Weinberger, K. Q. On Calibration of Modern Neural Networks;arXiv:1706.04599 [cs] (2017), which is incorporated herein in itsentirety by reference. After training, for each donor, we optimized thetemperature T on the nested cross-validation outer loop validation set.Then, we applied T to scale the Logit before Softmax computation andused the new Softmax score to infer classification confidence.

In addition to confidence calibration, we used dimension reduction toinvestigate the high-dimensional representations learned by ourpre-trained CNN models. Dimension reduction is a method to projecthigh-dimensional features into lower dimensions while preserving thecharacteristics of the data. Therefore, it provides a good way tovisualize how trained models represent different cell image inputs. Inour study, we choose UMAP (see, McInnes, L., Healy, J. & Melville, J.UMAP: Uniform Manifold Approximation and Projection for DimensionReduction; arXiv:1802.03426 [cs, stat] (2018); and McInnes, L., Healy,J., Saul, N. & GroBberger, L. UMAP: Uniform Manifold Approximation andProjection. J. Open Source Softw. 3, 861 (2018); DOI10.21105/joss.00861, both of which are incorporated herein in theirentirety by reference) as our dimension reduction algorithm. UMAP usesmanifold learning techniques to reduce feature dimensions. It arguablypreserves more of the global structure and is more scalable than thestandard form of t-SNE (see, Maaten, L. v. d. & Hinton, G. Visualizingdata using t-SNE. J. Mach. Learn. Res. 9, 2579-2605 (2008), which isincorporated herein in its entirety by reference), an alternativeapproach. Using UMAP, we projected the image features, extracted fromthe CNN layer right before the output layer, from 2048 dimensions to twodimensions. We used the default UMAP parameter values: “n neighbors” as15, “metrics” as “euclidean”, and “min dist” as 0:1. Then, we visualizedand analyzed these projected features of T cell images using 2D scatterplots. When comparing UMAP with t-SNE, we used the default t-SNEparameters: “perplexity” as 30 and “metric” as “euclidean”.

For the pre-trained CNN with fine-tuning, each test donor has differenttuned hyper-parameters and a different fine-tuned CNN. Therefore, weperformed feature extraction and dimension reduction independently foreach test donor. There is no guarantee that these five scatter plotsshare the same 2D basis. In contrast, the image pixel features,CellProfiler features, and off-the-shelf last layer features from apre-trained CNN do not vary by test donor. For these three UMAPapplications, we performed feature extraction and dimension reduction inone batch for all donors simultaneously. We excluded donor 4 from thedimension reduction analyses.

Finally, we used saliency maps to further analyze what morphologyfeatures were used in classification. See, Simonyan, K., Vedaldi, A. &Zisserman, A. Deep Inside Convolutional Networks: Visualising ImageClassification Models and Saliency Maps; arXiv:1312.6034 [cs](2013),which is incorporated herein in its entirety by reference. A saliencymap is a straightforward and efficient way to detect how predictionvalue changes with respect to a small change in the input cell imagepixels. It is generated by computing the gradient of the output classscore with respect to the input image. We compared two ways to computethis gradient: standard backpropagation and guided backpropagation. See,Springenberg, J. T., Dosovitskiy, A., Brox, T. & Riedmiller, M. Strivingfor Simplicity: The All Convolutional Net; arXiv:1412.6806 [cs](2014),which is incorporated herein in its entirety by reference.Backpropagation is a method to calculate the gradient of loss functionwith respect to the neural network's weights. Guided backpropagation isa variant that only backpropagates positive gradients. We generatedsaliency maps of the output layer for the pre-trained CNN withfine-tuning model for test donor 1 with a few randomly sampled imagesfrom the test set. The saliency map interpretations help us assesswhether the classification basis is intuitive and whether thepredictions derive from image artifacts instead of cell morphology.

Results: Overview

Our goal is to classify individual T cells as activated (positiveinstances) or quiescent (negative instances) using only croppedautofluorescence intensity cell images. We explore multipleclassification approaches of increasing complexity. A frequencyclassifier uses the frequency of positive samples in the training set asthe probability of the activated label. This naive baseline modelassesses how well the class skew in the training images can predict thelabel of new images. In addition, we test three Lasso logisticregression approaches on different featurizations of the cropped T cellimages. The first uses the image pixel intensities directly as features.The second uses only two image summaries as features, the cell size andtotal intensity. The third uses attributes calculated with CellProfiler,such as the mean intensity value and cell perimeter.

We also assess multiple types of neural networks. A fully connectedneural network (multilayer perceptron) generalizes the logisticregression model with pixel intensities by adding a single hidden layer.The LeNet CNN architecture learns convolutional filters that takeadvantage of the image structure of the input data. This CNN is simpleenough to train from random initialization with a limited number ofimages. Finally, we consider two deeper and more complex CNNs. Both usetransfer learning to initialize the Inception v3 CNN architecture with amodel that has been pre-trained on generic (nonbiological) images. Oneversion trains a new fully connected layer from scratch usingoff-the-shelf features extracted from cell images with the pre-trainedCNN. An alternative fine-tunes multiple layers of the pre-trained CNN.

The overall workflow for our pre-trained CNN model is described in FIG.4. The original microscopy images are segmented, cropped, and padded. Wefilter images that do not contain a T cell and other artifacts, leavingthe final image counts for each of the six donors shown in Table 3. Thenwe train, evaluate, and interpret the machine learning models. FIG. 4shows the training procedure for the pre-trained CNN as an example.

TABLE 3 Donor 1 Donor 2 Donor 3 Donor 4 Donor 5 Donor 6 InitialActivated 609 1139 604 789 791 531 Initial Quiescent 2184 399 2351 2110528 1007 Final Activated 235 647 446 482 683 442 Final Quiescent 1551141 1238 1569 246 580

The T cell microscopy images may vary from donor to donor. A trainedmodel must be able to generalize to new donors in order to be useful ina practical pre-clinical or clinical setting. Therefore, all of ourevaluation strategies train on images from some donors and evaluate thetrained models on separate images from a different donor, which isreferred to as subject-wise cross-validation or a leave-one-patient-outscheme. We initially assess the classifiers with cross-validation acrossdonors. In addition, we hold out all images from a randomly selecteddonor, donor 4, and only use them after completing the rest of our studyto confirm that our model selection and hyper-parameter tuningstrategies generalize to a new donor.

Results: Cross-Validation Across Donors

In order to assess our classifiers' performance on cell images from newdonors, we design a nested cross-validation scheme to train, tune, andtest all models. Due to this cross-validation design, the same modelcould have different optimal hyperparameters for different test donors.Therefore, we group the final model performance by test donors (FIG. 5).We plot multiple evaluation metrics because each metric rewardsdifferent behaviors. The area under the curve (AUC) and averageprecision are summary statistics of the receiver operatingcharacteristic (ROC) curve (FIG. 6) and precision recall (PR) curve(FIG. 7), respectively. For all three evaluation metrics, the twopre-trained CNN models outperform other classifiers.

The frequency classifier's average accuracy for all test donors is37.56% (FIG. 5 and Table 4). The low accuracy of this simple methodimplies that the majority class in the training and test sets is likelyto be different. For example, there are more activated cells from donor1 while there are more quiescent cells from the combination of donors 2,3, 5, and 6. This baseline establishes that classifiers that fail to usefeatures other than the label count will perform poorly.

TABLE 4 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 13.16% 13.16% 100.00% 13.16% 50.00% 235 1551 217.89% 0.00% 0.00% 82.11% 50.00% 647 141 3 73.52% 0.00% 0.00% 26.48%50.00% 446 1238 5 26.48% 0.00% 0.00% 78.52% 50.00% 683 246 6 56.75%0.00% 0.00% 43.25% 50.00% 442 580

Three logistic regression models using different features all givebetter classifications than the baseline model. Logistic regression withthe image pixel matrix leads to an average accuracy of 78.74% (FIG. 5and Table 5). Among those 6,724 pixel features, 5,822 features onaverage are removed by the Lasso regularization. To interpret thismodel, we plot the exponential of each pixel's coefficient to visualizethe odds ratios. This model learns the shape of cells (see, FIG. S1 ofAppendix A of U.S. Provisional Patent Application No. 62/886,139, whichis incorporated herein in its entirety by reference). Larger cells aremore likely to be classified as activated. Logistic regression usingonly mask size and total intensity as features gives slightly betterperformance with an average accuracy of 79.93% (FIG. 5 and Table 6). Forall test donors, the optimal coefficient of cell mask size is negative,whereas the coefficient of total intensity is positive. In practice, weexpect larger cells to be activated, but the negative coefficientindicates the model learns the wrong relationship of cell size andactivity state. This can be explained by the inconsistent cell sizedistribution across donors (see, FIG. S2 of Appendix A of U.S.Provisional Patent Application No. 62/886,139, which is incorporatedherein in its entirety by reference) and the correlation of cell sizeand total intensity (multicollinearity). Comparing the odds ratio of onestandard deviation increment of each feature, however, shows thislogistic regression model is much more sensitive to total intensity thancell size. Finally, the logistic regression model with CellProfilerattributes yields 87.14% average accuracy (FIG. 5 and Table 7). Aftercomputing the odds ratio adjusting to the standard deviation of eachfeature, attributes that are related to image intensity and cell areahave the strongest impact on the predictions.

TABLE 5 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 84.60% 43.79% 60.00% 53.90% 82.27% 235 1551 268.02% 94.99% 64.45% 95.38% 81.61% 647 141 3 79.57% 61.75% 60.09% 68.67%81.10% 446 1238 5 80.81% 88.12% 85.42% 95.20% 87.16% 683 246 6 80.68%73.43% 86.97% 90.19% 90.59% 442 580

TABLE 6 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 88.13% 55.35% 50.64% 53.67% 79.92% 235 1551 265.99% 96.56% 60.74% 94.46% 79.34% 647 141 3 81.59% 68.89% 55.61% 68.12%74.68% 446 1238 5 80.92% 89.32% 84.11% 95.20% 86.68% 683 246 6 83.02%78.17% 84.49% 89.86% 89.02% 442 580

TABLE 7 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 95.74% 86.98% 79.57% 88.85% 95.61% 235 1551 276.65% 91.56% 78.83% 95.24% 82.33% 647 141 3 92.16% 94.60% 74.66% 93.07%96.26% 446 1238 5 81.81% 81.81% 96.78% 93.74% 86.70% 683 246 6 89.33%82.33% 95.93% 95.97% 97.01% 442 580

Non-linear models with image pixels as input have accuracies comparableto the logistic regression model with CellProfiler features. We tune thelearning rate, batch size and the number of hidden layer neurons of thesimple neural network with one hidden layer. Even though its averageaccuracy of 86.48% (FIG. 5 and Table 8) is slightly lower than logisticregression with CellProfiler features, it has more stable performanceacross the test donors. In comparison, the LeNet CNN has a more complexarchitecture and takes advantage of the image structure of the inputdata. After selecting the best learning rate and batch size, LeNetreaches an average accuracy of 89.51% (FIG. 5 and Table 9).

TABLE 8 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 88.80% 55.28% 75.74% 65.40% 90.55% 235 1551 282.87% 96.05% 82.69% 96.60% 88.78% 647 141 3 88.54% 82.14% 72.20% 84.59%90.34% 446 1238 5 84.78% 85.58% 95.19% 96.48% 92.05% 683 246 6 87.41%80.62% 93.48% 94.44% 95.36% 442 580

TABLE 9 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 94.23% 78.21% 77.87% 82.15% 95.36% 235 1551 287.06% 96.58% 87.33% 97.86% 91.92% 647 141 3 91.45% 95.76% 70.85% 91.55%94.12% 446 1238 5 87.41% 87.83% 96.19% 96.40% 92.41% 663 246 6 87.38%78.61% 97.29% 96.70% 97.36% 442 580

Our most advanced models using the pre-trained CNN outperform all othermethods. Both versions of the pre-trained CNN use cell images as inputand require a previously trained CNN. For one version, we use thepre-trained CNN as a feature extractor and then train a new hidden layerwith off-the-shelf features. Alternatively, we fine-tune multiplehigher-level layers of the CNN with T cell images. We include thefine-tuned layers as a hyper-parameter. Specifically, we define n,ranging from 1 to 11, as the number of last Inception modules in thepre-trained Inception v3 CNN to fine-tune. For example, if n=1, we onlyfine-tune the last Inception module, whereas we fine-tune all the layersof the Inception v3 CNN when n=11. After tuning n along with the otherhyper-parameters, we compare the CNN with fine-tuning to the CNNoff-the-shelf model in order to study the effect of fine-tuning onclassifier performance. Additionally, we compare the test results ofdifferent n to analyze how the number of fine-tuned layers affectsclassification.

The average accuracy for the pre-trained CNN off-the-shelf model is90.36% (FIG. 5 and Table 10) and 93.56% for the pre-trained CNN withfine-tuning (FIG. 5 and Table 11). The fine-tuning model uses11;10;7;11; and 8 layers as the optimal n for the five test donors.However, depending on the test donor and the evaluation metric, thenumber of fine-tuned layers does not necessarily have a strong effect onthe predictive performance (FIG. 8). Different n values yield similarevaluation metrics. Fine-tuning all 11 layers also greatly increases theCNN training time (see, FIGS. S3 and S4 of Appendix A of U.S.Provisional Patent Application No. 62/886,139, which is incorporatedherein in its entirety by reference).

TABLE 10 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 94.57% 81.08% 76.60% 86.42% 95.96% 235 1551 290.10% 96.87% 90.88% 99.06% 95.89% 647 141 3 93.94% 93.22% 83.18% 94.08%96.66% 446 1238 5 87.08% 87.09% 96.78% 96.83% 92.81% 683 246 6 86.11%75.95% 99.32% 97.61% 98.49% 442 580

TABLE 11 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 1 96.81% 92.79% 82.13% 91.71% 95.70% 235 1551 291.88% 97.24% 92.74% 99.22% 97.09% 647 141 3 94.42% 96.32% 82.06% 95.75%97.41% 446 1238 5 89.77% 89.20% 97.95% 97.02% 94.26% 683 246 6 94.91%92.76% 95.70% 98.20% 98.91% 442 580

Results: Confirming Generalization with a New Donor

In order to evaluate our ability to generalize to T cell images from anew individual, we completely hold out images from donor 4 during thestudy design, model implementation, and cross-validation above. We applythe same nested cross-validation scheme to train, tune, and test thepre-trained CNN with fine-tuning, the most accurate model in theprevious cross-validation, on images from donor 4. It gives an accuracyof 98.83% (Table 12). Out of 2,051 predictions, there are only 4 falsepositives and 20 false negatives. The performance metrics in Table 12are substantially higher than their counterparts in Table 11. Havingtraining data from five donors instead of four likely contributes to theimproved performance.

TABLE 12 Donor Accuracy Precision Recall Average Precision AUC ActivatedCount Quiescent Count 4 98.83% 99.14% 95.85% 99.79% 99.93% 482 1569

Results: Pre-Trained CNN with Fine-Tuning Errors

We inspect the T cell images that the pre-trained CNN with fine-tuningclassifies incorrectly in order to better understand its failures andaccuracy. We visualize the misclassified images for all test donors(see, FIGS. S5-S10 of Appendix A of U.S. Provisional Patent ApplicationNo. 62/886,139, which is incorporated herein in its entirety byreference) along with the predicted label, the Softmax score of thenetwork output layer, and the temperature scaled confidence calibration.The majority of misclassified cell images are badly cropped, with nocells or multiple cells included in the frame. Therefore, using a moreprogressive dim image filter or adding a multiple-cell detector in theimage processing pipeline could further improve the model performance.However, for other images with a clear single cell in the frame, thepre-trained CNN tends to give high confidence in its misclassification.These scores suggest that these errors cannot be easily fixed without amore powerful classifier or more diverse training dataset. Temperaturescaling could either soften the original Softmax score toward 50% orincrease the confidence toward 100%. For the misclassified images in ourstudy, temperature scaling always drops the Softmax probability. Thisobservation matches Guo et al.'s finding that neural networks withhigher model capacity are more likely to be overconfident in theirpredictions.

Results: Pre-Trained CNN with Fine-Tuning Interpretation

Visualizing the T cell dataset in 2D helps us understand why someclassifiers perform better than others. We use Uniform ManifoldApproximation and Projection (UMAP) to project the images into 2D suchthat similar images in the original feature space are nearby in the 2Dspace. Coloring the images with their activity labels shows howdifferent input representations or learned representations separate theactivated and quiescent cells. For example, in FIG. 9, each dotcorresponds to one image based on its representation in the last layerof the pre-trained CNNs with fine-tuning. UMAP projects the 2048 learnedfeatures in the last layer of the CNN into 2D. In general, the activatedand quiescent cells are well-separated in the 2D space, suggesting thatthe CNN has successfully learned distinct representations for the twotypes of T cells. Using t-Distributed Stochastic Neighbor Embedding(t-SNE) instead of UMAP for dimension reduction provides qualitativelysimilar results (see, FIG. S11 of Appendix A of U.S. Provisional PatentApplication No. 62/886,139, which is incorporated herein in its entiretyby reference).

Generating similar UMAP plots for three alternative imagerepresentations shows that the two image classes are not as wellseparated (see, FIGS. S12-S14 of Appendix A of U.S. Provisional PatentApplication No. 62/886,139, which is incorporated herein in its entiretyby reference). When using the raw pixel features (see, FIG. S12 ofAppendix A of U.S. Provisional Patent Application No. 62/886,139, whichis incorporated herein in its entirety by reference), the two types of Tcells are spread throughout the 2D space. This contributes to the lowerperformance of the logistic regression and fully connected neuralnetwork models that operate directly on pixel intensity. Similarly,there is only moderate spatial separation when using the CellProfilerfeatures (see, FIG. S13 of Appendix A of U.S. Provisional PatentApplication No. 62/886,139, which is incorporated herein in its entiretyby reference) or the last layer of the CNN before fine-tuning it topredict T cell activity (see, FIG. S14 of Appendix A of U.S. ProvisionalPatent Application No. 62/886,139, which is incorporated herein in itsentirety by reference). These comparisons demonstrate the strong effectof fine-tuning the pre-trained CNN and also help explain the superiorperformance of pre-trained CNNs over the logistic regression model withCellProfiler features. In addition, by labeling misclassified images asoutlined dots in FIG. 9, we see where the pre-trained CNN withfine-tuning makes errors. The incorrect predictions are predominantlydistributed in the boundary between the two clusters.

In addition to visualizing the feature representation in the pre-trainedCNNs with fine-tuning, we use saliency maps to interpret how thesemodels make decisions. We generate saliency maps by computing thegradient of the CNN class score with respect to a few randomly chosendonor 1 images from both the activated and quiescent classes (FIG. 10).We use two methods to calculate gradients: standard backpropagation andguided backpropagation. In these heat maps, larger values (green oryellow) highlight the image regions that cause the most change in the Tcell activity prediction. Smaller values (dark blue or purple) indicatepixels that have less influence. The uniformly dark blue background inboth types of saliency maps indicates that the pre-trained CNNs withfine-tuning have learned to focus on the original cell image instead ofthe black padding. The larger values in the saliency maps with guidedbackpropagation often align with the high-intensity regions of the cellimages, which correspond to mitochondria and depict metabolic activity.Although the influential regions of the guided backpropagation-basedsaliency maps are biologically plausible, this type of saliency map isinsensitive to random changes of either the input data or modelparameters. The saliency maps generated with standard backpropagationare properly affected by these randomized controls but do notconcentrate on the high-intensity regions of the input images.

DISCUSSION

Our study demonstrates that machine learning models trained onautofluorescence intensity images can accurately classify activated andquiescent T cells across donors. Because autofluorescence images areeasier to acquire with standard commercial microscopes compared tofluorescence lifetime images, this workflow has the potential to becomea widely applicable approach for live T cell profiling. Fine-tuning apre-trained CNN is the most powerful classification approach,outperforming alternative machine learning models that are commonly usedfor microscopy image classification over multiple evaluation metrics. Inparticular, this CNN applied directly to cropped images has betterperformance than logistic regression with domain-relevant featuresextracted by CellProfiler.

We thoroughly explored the effect of fine-tuning more layers of thepre-trained CNN and compared it with the off-the-shelf CNN model. Thecommon transfer learning approach fixes the CNN parameters of theinitial network layers, which extract learned features from the images,and trains a simple classifier from scratch that predicts thedomain-specific image labels. Our results indicate that fine-tuningpre-trained CNN layers yields better performance than directly usingoff-the-shelf features. In addition, although fine-tuning more layerstends to give better predictive performance (FIG. 8), it is generallynot worth the additional computational time and expense to fine-tune all11 layers (see, FIGS. S3 and S4 of Appendix A of U.S. Provisional PatentApplication No. 62/886,139, which is incorporated herein in its entiretyby reference). Possible reasons include the limited sample size andrelatively homogeneous cell image representations. Given the extracomputational costs and implementation challenges, we recommendfine-tuning only the last few layers of a pre-trained CNN for similarautofluorescence microscopy applications. In settings that do requirefine-tuning additional layers because the images are more heterogeneous,we suggest taking a larger step size in the layer number hyper-parameteroptimization.

The machine learning models recognize image attributes that recapitulatebiological domain knowledge. Activated T cells are larger in size. Inaddition, there are metabolic differences between quiescent andactivated T cells, which are evident in the NAD(P)H images. The highintensity regions in the images likely correspond to mitochondria, wherethe majority of metabolism occurs. It is straightforward to inspect thetrained logistic regression model that takes total image intensity andmask size as inputs and observe that it correctly recognizes therelationship between NAD(P)H intensity and activation state.

The parameters of the pre-trained CNN with fine-tuning are not asdirectly interpretable as the logistic regression model. An additionalchallenge is that different interpretation techniques provide distinctviews of the fine-tuned CNN. Nevertheless, there are some indications inthe saliency maps that this CNN also reflects T cell biology. Saliencymaps help locate which regions of the input image influence theclassification the most. With guided backpropagation, the high-intensityregions of the T cell images tend to be the focal points in the saliencymaps. This suggests that the CNN may be sensitive to metabolicdifferences between quiescent and activated cells and not only changesin cell size. However, guided backpropagation and other more advancedsaliency maps were found to be independent of the data, model, and modelparameters. The standard backpropagation gradient map is sensitive tothese controls, but it focuses more on general cell morphology than themetabolic activity within cells.

Each model in our study is only tuned and evaluated once, which limitsour ability to assess the statistical significance of the performancedifferences across models. Substantial computing time and costs arerequired for nested cross-validation, especially when fine-tuningmultiple layers of the pre-trained CNN (see, FIGS. S3 and S4 of AppendixA of U.S. Provisional Patent Application No. 62/886,139, which isincorporated herein in its entirety by reference). The fine-tuning jobstook 5,096 hours (212 days) in total to train on GPUs. Therefore, we areunable train each model multiple times to assess the variability inmodel performance due to random sampling, computer hardware,non-deterministic algorithms, and other factors. Slight differences inperformance should not be over-interpreted.

Based on the misclassified images, the performance of the pre-trainedCNN model with fine-tuning is limited by the image cropping quality.Some images contain multiple cells. Others do not contain any T cells.Developing a better filter to detect images with artifacts and adoptingstate-of-the-art segmentation approaches could further boostclassification accuracy.

Overall, our strong results demonstrate the feasibility of classifying Tcells directly from autofluorescence intensity images, which can guidefuture work to bring this technology to pre-clinical and clinicalapplications.

The present disclosure also includes the following statements:

1. A T cell classifying device comprising:

a cell analysis pathway comprising:

-   -   (i) an inlet;    -   (ii) an observation zone coupled to the inlet downstream of the        inlet, the observation zone configured to present T cells for        individual autofluorescence interrogation; and    -   (iii) an outlet coupled to the observation zone downstream of        the observation zone;

a single-cell autofluorescence image sensor configured to acquire anautofluorescence intensity image of a T cell positioned in theobservation zone;

a processor in electronic communication with the single-cellautofluorescence image sensor; and

a non-transitory computer-readable medium accessible to the processorand having stored thereon a trained convolutional neural network andinstructions that, when executed by the processor, cause the processorto:

-   -   a) receive the autofluorescence intensity image;    -   b) optionally pre-process the autofluorescence intensity image        to produce an adjusted autofluorescence intensity image;    -   c) input the autofluorescence intensity image or the adjusted        autofluorescence intensity image into the trained convolutional        neural network to produce an activation prediction for the T        cell.        2. The T cell classifying device of statement 1, wherein the        cell analysis pathway comprises a microfluidic pathway or a        nanofluidic pathway.        3. The T cell classifying device of statement 1 or 2, the T cell        classifying device further comprising a flow regulator coupled        to the inlet.        4. The T cell classifying device of any one of the preceding        statements, wherein the flow regulator is configured to provide        flow of cells through the observation zone at a rate that allows        the single-cell autofluorescence imager to acquire the        autofluorescence image for the T cell when it is positioned in        the observation zone.        5. The T cell classifying device of any one of the preceding        statements, the T cell classifying device further comprising a        light source.        6. The T cell classifying device of any one of the preceding        statements, wherein the light source is a continuous wave light        source.        7. The T cell classifying device of any one of the preceding        statements, wherein the light source emits light having a        wavelength tuned to excite fluorescence from NAD(P)H and/or FAD.        8. The T cell classifying device of any one of the preceding        statements, wherein the single-cell autofluorescence image        sensor is configured to acquire the autofluorescence intensity        image at a repetition rate of between 1 kHz and 20 GHz.        9. The T cell classifying device of any one of the preceding        statements, wherein the single-cell autofluorescence image        sensor is configured to acquire the autofluorescence intensity        image at a repetition rate of between 1 MHz and 1 GHz.        10. The T cell classifying device of any one of the preceding        statements, wherein the single-cell autofluorescence image        sensor is configured to acquire the autofluorescence intensity        image at a repetition rate of 20 MHz and 100 MHz.        11. The T cell classifying device of any one of the preceding        statements, wherein the single-cell autofluorescence image        sensor is configured to acquire the autofluorescence intensity        image via pixel-by-pixel intensity measurements.        12. The T cell classifying device of any one of the preceding        statements, wherein the single-cell autofluorescence image        sensor is a charge collection device array.        13. The T cell classifying device of any one of the preceding        statements, the single-cell autofluorescence image sensor        comprising a detector-side filter configured to transmit        fluorescence signals of interest.        14. The T cell classifying device of the immediately preceding        statement, wherein the detector-side filter is configured to        transmit NAD(P)H fluorescence and/or FAD fluorescence.        15. The T cell classifying device of any one of the preceding        statements, the T cell classifying device further comprising a        cell size measurement tool configured to measure cell size and        to communicate the cell size to the processor.        16. The T cell classifying device of any one of the preceding        statements, the T cell classifying device further comprising a        cell imager configured to acquire an image of a cell positioned        within the observation zone and to communicate the image to the        processor.        17. The T cell classifying device of any one of the preceding        statements, wherein the instructions, when executed by the        processor, cause the processor to: b) pre-process the        autofluorescence intensity image to produce an adjusted        autofluorescence intensity image; and c) input the adjusted        autofluorescence intensity image into the trained convolutional        neural network to produce the activation prediction for the T        cell.        18. The T cell classifying device of the immediately preceding        statement, wherein the pre-processing of step b) includes        cropping the autofluorescence intensity image, padding the        autofluorescence intensity image, rotating the autofluorescence        intensity image, reflecting the autofluorescence intensity        image, or a combination thereof.        19. The T cell classifying device of any one of the preceding        statements, wherein the instructions, when executed by the        processor, cause the processor to determine if the        autofluorescence intensity image is an outlier and to skip steps        b), c) and d) if the autofluorescence intensity image is an        outlier.        20. The T cell classifying device of any one of the preceding        statements, wherein the cell analysis pathway does not include a        fluorescent label for binding to the T cell.        21. The T cell classifying device of any one of the preceding        statements, wherein the autofluorescence image sensor is adapted        to measure autofluorescence of T cells without requiring the use        of a fluorescent label.        22. The T cell classifying device of any one of the preceding        statements, wherein the cell analysis pathway does not include        an immobilization agent for binding and immobilizing T cells.        23. The T cell classification device of any one of the preceding        statements, the T cell classification device further comprising        a cell sorter having a sorter inlet and at least two sorter        outlets, the cell sorter coupled to the cell analysis pathway        via the outlet downstream of the observation zone, the cell        sorter configured to selectively direct a cell from the sorter        inlet to one of the at least two sorter outlets based on a sort        signal, the processor in electronic communication with the cell        sorter, and the instructions, when executed by the processor,        further cause the processor to provide the sort signal to the        cell sorter based on the activation prediction.        24. The T cell classifying device of the immediately preceding        statement, wherein the trained convolutional neural network, the        processor, and physical dimensions and flow rate of the cell        analysis pathway are adapted to provide the sorter signal to the        cell sorter prior to the T cell reaching the cell sorter.        25. The T cell classifying device of any one of the preceding        claims, wherein the instructions, when executed by the        processor, further cause the processor to generate a report        including the activation prediction for T cells having passed        through the cell analysis pathway.        26. A method of characterizing T cell activation state, the        method comprising:

a) optionally receiving a population of T cells having unknownactivation status;

b) acquiring an autofluorescence intensity image for a T cell of thepopulation of T cells;

c) optionally pre-processing the autofluorescence intensity image toprovide an adjusted autofluorescence intensity image; and

d) identifying an activation status of the T cell based on an activationprediction, wherein the activation prediction is computed using theautofluorescence intensity image or the adjusted autofluorescenceintensity image as an input for a trained convolutional neural network.

27. A method of classifying T cells, the method comprising:

a) receiving a population of T cells having unknown activation status;

b) acquiring an autofluorescence intensity image for each T cell of thepopulation of T cells, thereby resulting in a set of autofluorescenceintensity images;

c) optionally pre-processing the autofluorescence intensity images ofthe set of autofluorescence intensity images to provide a set ofadjusted autofluorescence intensity images; and

either:

d1) physically isolating a first portion of the population of T cellsfrom a second portion of the population of T cells based on anactivation prediction, wherein each T cell of the population of T cellsis placed into the first portion when the activation prediction exceedsa predetermined threshold and into the second portion when theactivation prediction is less than or equal to the predeterminedthreshold; or

d2) generating a report including the activation prediction, the reportoptionally identifying a proportion of the population of T cells havingthe activation prediction that exceeds the predetermined threshold,

wherein the activation prediction is computed using the autofluorescenceintensity image from the set of autofluorescence intensity images or theadjusted autofluorescence intensity image from the set of adjustedautofluorescence intensity images corresponding to a given T cell as aninput for a trained convolutional neural network.28. The T cell classifying device or the method of any one of thepreceding statements, wherein the autofluorescence intensity image istuned to a wavelength corresponding to NAD(P)H fluorescence and/or FADfluorescence.29. The T cell classifying device or the method of any one of thepreceding statements, wherein at least a portion of the trainedconvolutional neural network is initially pre-trained using images thatare not fluorescence images of cells.30. The T cell classifying device or the method of any one of thepreceding statements, wherein at least a portion of the trainedconvolutional neural network includes an image classification network atleast partially pre-trained using optical images of objects that arevisible to the naked human eye.31. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkutilizes a spatial distribution of fluorescence intensity in producingthe activation prediction.32. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkis trained using only fluorescence intensity images as an input.33. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkis not pre-trained or trained with a cell size attribute as an input anddoes not use the cell size attribute as an input to produce theactivation prediction.34. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkis not pre-trained or trained with cell morphological features as aninput and does not use cell morphological features as an input toproduce the activation prediction.35. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkis trained on at least 100 images, at least 500 images, at least 1000images, at least 2500 images, or at least 5000 images of T cells havingknown activation states.36. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networksegments the autofluorescence intensity image.37. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkis instrument-specific.38. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkis patient-specific.39. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkprovides a classification accuracy of at least 85%, at least 87.5%, atleast 90%, at least 92.5%, at least 95%, at least 96%, at least 97%, orat least 98%.40. The T cell classifying device or the method of any one of thepreceding statements, wherein the trained convolutional neural networkincludes at least 5 layers, at least 6 layers, at least 7 layers, atleast 8 layers, at least 9 layers, or at least 10 layers, and at most100 layers, at most 50 layers, at most 20 layers, at most 17 layers, atmost 15 layers, at most 14 layers, at most 13 layers, or at most 12layers.41. The T cell classifying device or the method of any one of thepreceding statements, wherein the T cells whose activation prediction ispositive are CD3+, CD4+ or CD8+ T cells.42. The method of any one of statements 26 to the immediately precedingstatement, wherein step c) is not optional and the activation predictionof step d) is computed using the adjusted autofluorescence intensityimage.43. The method of the immediately preceding claim, wherein thepre-processing of step c) includes cropping the autofluorescenceintensity image, padding the autofluorescence intensity image, rotatingthe autofluorescence intensity image, reflecting the autofluorescenceintensity image, or a combination thereof.44. The method of any one of statements 26 to the immediately precedingstatement, the method further comprising determining if theautofluorescence intensity image is an outlier and skipping step d) ifthe autofluorescence intensity image is an outlier.45. The method of any one of statements 26 to the immediately precedingstatement, wherein the method does not involve use of a fluorescentlabel for binding to the T cell.46. The method of any one of statements 26 to the immediately precedingstatement, wherein the method does not involve immobilizing the T cell.47. A method of administering activated T cells to a subject in needthereof, the method comprising:

a) the method of any one of statements 27 to the immediately precedingstatement, wherein the method comprises step d1); and

b) introducing the first portion of the population of T cells to thesubject.

48. The method of statement 47, wherein the first portion of thepopulation of T cells is modified prior to step b).

49. The method of statement 48, wherein the first portion of thepopulation of T cells is modified to include a chimeric antigen receptorprior to step b).

50. A method of administering activated T cells to a subject in needthereof, the method comprising:

a) the method of any one of statements 27 to 46, wherein the methodcomprises step d2); and

b) in response to the proportion exceeding a second predeterminedthreshold, introducing the population of T cells to the subject.

51. The method of the immediately preceding statement, wherein thepopulation of T cells is modified prior to step b).

52. The method of the immediately preceding statement, wherein thepopulation of T cells is modified to include a chimeric antigen receptorprior to step b).

We claim:
 1. A T cell classifying device comprising: a cell analysispathway comprising: (i) an inlet; (ii) an observation zone coupled tothe inlet downstream of the inlet, the observation zone configured topresent T cells for individual autofluorescence interrogation; and (iii)an outlet coupled to the observation zone downstream of the observationzone; a single-cell autofluorescence image sensor configured to acquirean autofluorescence intensity image of a T cell positioned in theobservation zone; a processor in electronic communication with thesingle-cell autofluorescence image sensor; and a non-transitorycomputer-readable medium accessible to the processor and having storedthereon a trained convolutional neural network and instructions that,when executed by the processor, cause the processor to: a) receive theautofluorescence intensity image; b) optionally pre-process theautofluorescence intensity image to produce an adjusted autofluorescenceintensity image; c) input the autofluorescence intensity image or theadjusted autofluorescence intensity image into the trained convolutionalneural network to produce an activation prediction for the T cell. 2.The T cell classifying device of claim 1, wherein the autofluorescenceintensity image is tuned to a wavelength corresponding to NAD(P)Hfluorescence and/or FAD fluorescence.
 3. The T cell classifying deviceof claim 1, wherein at least a portion of the trained convolutionalneural network is initially pre-trained using images that are notfluorescence images of cells.
 4. The T cell classifying device of claim1, wherein at least a portion of the trained convolutional neuralnetwork includes an image classification network at least partiallypre-trained using optical images of objects that are visible to thenaked human eye.
 5. The T cell classifying device of claim 1, whereinthe trained convolutional neural network utilizes a spatial distributionof fluorescence intensity in producing the activation prediction.
 6. TheT cell classifying device of claim 1, wherein the trained convolutionalneural network is trained using only fluorescence intensity images as aninput.
 7. The T cell classifying device of claim 1, wherein the trainedconvolutional neural network is not pre-trained or trained with a cellsize attribute as an input and does not use the cell size attribute asan input to produce the activation prediction.
 8. The T cell classifyingdevice of claim 1, wherein the trained convolutional neural network isnot pre-trained or trained with cell morphological features as an inputand does not use cell morphological features as an input to produce theactivation prediction.
 9. The T cell classifying device of claim 1,wherein the trained convolutional neural network segments theautofluorescence intensity image.
 10. The T cell classifying device ofclaim 1, wherein the trained convolutional neural network provides aclassification accuracy of at least 85%.
 11. The T cell classifyingdevice of claim 1, wherein the instructions, when executed by theprocessor, cause the processor to: b) pre-process the autofluorescenceintensity image to produce an adjusted autofluorescence intensity image;and c) input the adjusted autofluorescence intensity image into thetrained convolutional neural network to produce the activationprediction for the T cell.
 12. The T cell classifying device of claim11, wherein the pre-processing of step b) includes cropping theautofluorescence intensity image, padding the autofluorescence intensityimage, rotating the autofluorescence intensity image, reflecting theautofluorescence intensity image, or a combination thereof.
 13. The Tcell classifying device of claim 1, wherein the instructions, whenexecuted by the processor, cause the processor to determine if theautofluorescence intensity image is an outlier and to skip steps b), c)and d) if the autofluorescence intensity image is an outlier.
 14. Amethod of classifying T cells, the method comprising: a) receiving apopulation of T cells having unknown activation status; b) acquiring anautofluorescence intensity image for each T cell of the population of Tcells, thereby resulting in a set of autofluorescence intensity images;c) optionally pre-processing the autofluorescence intensity images ofthe set of autofluorescence intensity images to provide a set ofadjusted autofluorescence intensity images; and either: d1) physicallyisolating a first portion of the population of T cells from a secondportion of the population of T cells based on an activation prediction,wherein each T cell of the population of T cells is placed into thefirst portion when the activation prediction exceeds a predeterminedthreshold and into the second portion when the activation prediction isless than or equal to the predetermined threshold; or d2) generating areport including the activation prediction, the report optionallyidentifying a proportion of the population of T cells having theactivation prediction that exceeds the predetermined threshold, whereinthe activation prediction is computed using the autofluorescenceintensity image from the set of autofluorescence intensity images or theadjusted autofluorescence intensity image from the set of adjustedautofluorescence intensity images corresponding to a given T cell as aninput for a trained convolutional neural network.
 15. The method ofclaim 14, wherein step c) is performed and the activation prediction ofstep d1) or step d2) is computed using the adjusted autofluorescenceintensity image.
 16. The method of claim 15, wherein the pre-processingof step c) includes cropping the autofluorescence intensity image,padding the autofluorescence intensity image, rotating theautofluorescence intensity image, reflecting the autofluorescenceintensity image, or a combination thereof.
 17. The method of claim 14,the method further comprising determining if the autofluorescenceintensity image is an outlier and skipping both step d1) and step d2) ifthe autofluorescence intensity image is an outlier.
 18. The method ofclaim 14, wherein the autofluorescence intensity image is tuned to awavelength corresponding to NAD(P)H fluorescence and/or FADfluorescence.
 19. The method of claim 14, wherein at least a portion ofthe trained convolutional neural network is initially pre-trained usingimages that are not fluorescence images of cells.
 20. A method ofadministering activated T cells to a subject in need thereof, the methodcomprising: the method of claim 14; and introducing the first portion ofthe population of T cells to the subject.